In [16]:
__author__ = 'Brian Merino <brian.merino@noirlab.edu>, Vinicius Placco <vinicius.placco@noirlab.edu>'
__version__ = '20240606' # yyyymmdd; version datestamp of this notebook
__keywords__ = ['gsaoi','gemini','galaxy','dragons']

Gemini GSAOI elliptical galaxy reduction using DRAGONS Python API¶

Public archival data from gsaoiimg_tutorial - GS-2017A-Q-29 (NGC5128)¶

adapted from https://dragons.readthedocs.io/projects/gsaoiimg-drtutorial/en/v3.1.0/index.html¶

Disclaimer: Due to the large size of GSAOI images, this notebook will likely take 20+ minutes to finish running.¶

Table of contents¶

  • Goals
  • Summary
  • Disclaimers and attribution
  • Imports and setup
  • About the dataset
  • Downloading data for reduction
  • Set up the DRAGONS logger
  • Create Master Flat Field
  • Create File Lists
  • Standard Star
  • Reduce Science Images
  • Display stacked final image
  • Clean-up (optional)

Goals¶

Showcase how to reduce GSAOI imaging data using the Gemini DRAGONS package on the Data Lab science platform using a custom DRAGONS kernel "DRAGONS (Py3.7)". The steps include downloading data from the Gemini archive, setting up a DRAGONS calibration service, processing flats, darks, a bad pixel mask, and science frames, and creating a single combined stacked image.

Summary¶

DRAGONS is a Python-based astronomical data reduction platform written by the Gemini Science User Support Department. It can currently be used to reduce imaging data from Gemini instruments GMOS, NIRI, Flamingos 2, GSAOI, and GNIRS, as well as spectroscopic data in GMOS longslit mode. Linked here is a general list of guides, manuals, and tutorials about the use of DRAGONS.

The DRAGONS kernel has been made available in the Data Lab environment, allowing users to access the routines without being dependent on installing the software on their local machines.

In this notebook, we present an example of a DRAGONS Jupyter notebook that works in the Data Lab environment to reduce example Gemini South NIRI Kshort-band imaging data fully. This notebook will not present all of the details of the many options available to adjust or optimize the DRAGONS GSAOI data reduction process; rather, it will just show one example of a standard reduction of a GSAOI imaging dataset.

The data used in this notebook example is GSAOI KshortH-band imaging from the Gemini archive of the giant elliptical galaxy NGC 5128 from the Gemini South program "Stellar Population GeMology: Long Period Variables at High Metallicity in the Nearest Elliptical Galaxy", PI: John Blakeslee, program ID GS-2017A-Q-29.

Disclaimer & attribution¶

Disclaimers¶

Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.

Acknowledgments¶

If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:

This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:

This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

In either case please cite the following papers:

  • Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445

  • Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411

If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:

  • Juneau et al., "Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097

If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}

And if you are using SPARCL, please also add \software{SPARCL} and cite:

  • Juneau et al., "SPARCL: SPectra Analysis and Retrievable Catalog Lab", Conference Proceedings for ADASS XXXIII, 2024

https://doi.org/10.48550/arXiv.2401.05576

The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.

For this notebook specifically, please acknowledge:

  • DRAGONS publication: Labrie et al., "DRAGONS - Data Reduction for Astronomy from Gemini Observatory North and South", ASPC, 523, 321L

  • DRAGONS open source software publication

Importing python libraries¶

In [2]:
import glob

from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce

from gempy.adlibrary import dataselect
from gempy.utils import logutils

from astropy.io import fits
from astropy.wcs import WCS

import matplotlib.pyplot as plt
from matplotlib.colors import Normalize

About the dataset¶

The data is a GSAOI observation of the resolved outskirt of a nearby galaxy. The observation is a dither-on-target with offset-to-sky sequence.

The calibrations we use in this example are:

  • BPM. The bad pixel masks are now found in the Gemini Science Archive instead of being packaged with the software. They are associated like the other calibrations. Here we need the 2x2 binning GMOS-N Hamamatsu BPM valid for data taken in 2017. (The date in the name is the "valid from" date.)
  • Flats, as a sequence of lamps-on and lamps-off exposures.
  • A Standard star that could be used for photometry.
Observation Type File name(s) Purpose and Exposure (seconds)
Science S20170505S0095-110 Kshort-band, on target, 60 s
Flats S20170505S0030-044 Lamp on, Kshort, for science
Flats S20170505S0060-074 Lamp off, Kshort, for science
Science darks S20170504S0114-117 Kshort, standard star, 30 s

Note that for GSAOI, the dark current is low enough that there is no need to correct for it.

Downloading the data¶

Downloading Kshort-band images from the Gemini archive to the current working directory. This step only needs to be executed once.

If you run this notebook for the first time and need to download the dataset, set the variable "download=True". The notebook will not redownload the dataset if it is set to False. This will become particularly useful if you run the notebooks more than once.

In [3]:
%%bash 

# create file that lists FITS files to be downloaded
echo "\
http://archive.gemini.edu/file/S20170505S0095.fits
http://archive.gemini.edu/file/S20170505S0096.fits
http://archive.gemini.edu/file/S20170505S0097.fits
http://archive.gemini.edu/file/S20170505S0098.fits
http://archive.gemini.edu/file/S20170505S0099.fits
http://archive.gemini.edu/file/S20170505S0100.fits
http://archive.gemini.edu/file/S20170505S0101.fits
http://archive.gemini.edu/file/S20170505S0102.fits
http://archive.gemini.edu/file/S20170505S0103.fits
http://archive.gemini.edu/file/S20170505S0104.fits
http://archive.gemini.edu/file/S20170505S0105.fits
http://archive.gemini.edu/file/S20170505S0106.fits
http://archive.gemini.edu/file/S20170505S0107.fits
http://archive.gemini.edu/file/S20170505S0108.fits
http://archive.gemini.edu/file/S20170505S0109.fits
http://archive.gemini.edu/file/S20170505S0110.fits
http://archive.gemini.edu/file/S20170505S0030.fits
http://archive.gemini.edu/file/S20170505S0031.fits
http://archive.gemini.edu/file/S20170505S0032.fits
http://archive.gemini.edu/file/S20170505S0033.fits
http://archive.gemini.edu/file/S20170505S0034.fits
http://archive.gemini.edu/file/S20170505S0035.fits
http://archive.gemini.edu/file/S20170505S0036.fits
http://archive.gemini.edu/file/S20170505S0037.fits
http://archive.gemini.edu/file/S20170505S0038.fits
http://archive.gemini.edu/file/S20170505S0039.fits
http://archive.gemini.edu/file/S20170505S0040.fits
http://archive.gemini.edu/file/S20170505S0041.fits
http://archive.gemini.edu/file/S20170505S0042.fits
http://archive.gemini.edu/file/S20170505S0043.fits
http://archive.gemini.edu/file/S20170505S0044.fits
http://archive.gemini.edu/file/S20170505S0060.fits
http://archive.gemini.edu/file/S20170505S0061.fits
http://archive.gemini.edu/file/S20170505S0062.fits
http://archive.gemini.edu/file/S20170505S0063.fits
http://archive.gemini.edu/file/S20170505S0064.fits
http://archive.gemini.edu/file/S20170505S0065.fits
http://archive.gemini.edu/file/S20170505S0066.fits
http://archive.gemini.edu/file/S20170505S0067.fits
http://archive.gemini.edu/file/S20170505S0068.fits
http://archive.gemini.edu/file/S20170505S0069.fits
http://archive.gemini.edu/file/S20170505S0070.fits
http://archive.gemini.edu/file/S20170505S0071.fits
http://archive.gemini.edu/file/S20170505S0072.fits
http://archive.gemini.edu/file/S20170505S0073.fits
http://archive.gemini.edu/file/S20170505S0074.fits
http://archive.gemini.edu/file/S20170504S0114.fits
http://archive.gemini.edu/file/S20170504S0115.fits
http://archive.gemini.edu/file/S20170504S0116.fits
http://archive.gemini.edu/file/S20170504S0117.fits\
" > gsaoi.list
In [4]:
%%bash

download="True"

if [ $download == "True" ]; then
    wget --no-check-certificate -N -q -i gsaoi.list

else
    echo "Skipping download. To download the data set used in this notebook, set download=True."
fi

Setting up the DRAGONS logger¶

DRAGONS comes with a local calibration manager that uses the same calibration association rules as the Gemini Observatory Archive. This allows reduce to make requests to a local light-weight database for matching processed calibrations when needed to reduce a dataset.

This tells the system where to put the calibration database. This database will keep track of the processed calibrations we will send to it.

In [5]:
logutils.config(file_name='gsaoi_data_reduction.log')
caldb = cal_service.set_local_database()
caldb.init("w")

Create a list of all the FITS files in the directory

In [6]:
all_files = glob.glob('S2017*[0-9].fits')
all_files.sort()

Create file lists¶

The first step is to create input file lists. The tool "dataselect" helps with that. It uses Astrodata tags and "descriptors" to select the files and send the filenames to a text file that can then be fed to "reduce". (See the Astrodata User Manual for information about Astrodata.)

In [7]:
list_of_flats_Ks = dataselect.select_data(
     all_files,['FLAT'],[],
     dataselect.expr_parser('filter_name=="Kshort"'))

list_of_std_stars = dataselect.select_data(
    all_files,[],[],
    dataselect.expr_parser('observation_class=="partnerCal"'))

list_of_science_images = dataselect.select_data(
    all_files,[],[],
    dataselect.expr_parser('(observation_class=="science" and exposure_time==60.)'))

Create master flat field¶

The GSAOI Kshort master flat is created from a series of lamp-on and lamp-off dome exposures. They should all have the same exposure time. Each flavor is stacked (averaged), then the lamp-off stack is subtracted from the lamp-on stack, and the result is normalized.

The master flat will be saved with the suffix _flat.fits.

In [8]:
reduce_flats = Reduce()
reduce_flats.files.extend(list_of_flats_Ks)
reduce_flats.runr()

All submitted files appear valid:
S20170505S0030.fits ... S20170505S0074.fits, 30 files submitted.

Ginga not installed, use other viewer, or no viewer

Reduce standard star¶

The standard star is reduced essentially like the science target (next section). The processed flat field we added to the local calibration database will be fetched automatically. Also, in this case, the standard star was obtained using ROIs (Regions-of-Interest), which do not match the flat field. The software will recognize that the flat field is valid and crop it to match the ROIs.

This cell will create three files for each of the science darks. They will contain the suffixes _countsScaled.fits, _flatCorrected.fits, and sourcesDetected.fits.

NOTE: After running this cell, you may receive WARNINGS that there is no REFCAT available for some files or ERRORS regarding the extensions having different shapes. These WARNINGS and ERRORS will not affect the final product and can be ignored.

In [9]:
reduce_std = Reduce()
reduce_std.files.extend(list_of_std_stars)
reduce_std.runr()

   ERROR - Extensions have different shapes
   ERROR - Extensions have different shapes
   ERROR - Extensions have different shapes
   ERROR - Extensions have different shapes
   WARNING - No REFCAT in S20170504S0114_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170504S0115_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170504S0116_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170504S0117_sourcesDetected.fits - cannot calculate astrometry
   WARNING - Downgrading fit to order 2 due to limited number of matches.
   WARNING - Downgrading fit to order 1 due to limited number of matches.

Reduce science images¶

This is an observation of a galaxy with offset to sky. We need to turn off the additive offsetting of the sky because the target fills the field of view and does not represent a reasonable sky background. If the offsetting is not turned off in this particular case, it results in an over-subtraction of the sky frame.

Note: Unlike the other near-IR instruments, the additive offset_sky parameter is used by default to adjust the sky frame background for GSAOI instead of the multiplicative scale_sky parameter. It was found to work better when the sky background per pixel is very low. This is common due to the short exposure time needed to avoid saturating stars and the small pixel scale. The reader is encouraged to experiment with scale_sky if offset_sky does not seem to lead to an optimal sky subtraction.

(Remember that both parameters normally must be turned off when the source is extended.)

The sky frame comes from off-target sky observations. We feed the pipeline all the on-target and off-target frames. The software will appropriately split the on-target and off-target using information in the headers.

This command will generate flat-corrected and sky-subtracted files but will not stack them. You can differentiate each file by its suffix (_flatCorrected or _skyCorrected). The on-target files are the ones that have been sky subtracted (_skyCorrected). Finally, the stacked image will be saved with the suffix _image.fits.

NOTE: After running this cell, you may receive WARNINGS that there is no REFCAT available for some files. These WARNINGS will not affect the final product and can be ignored.

In [10]:
reduce_target = Reduce()
reduce_target.files.extend(list_of_science_images)
reduce_target.uparms.append(('skyCorrect:offset_sky', False))
reduce_target.runr()

   WARNING - No REFCAT in S20170505S0095_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0096_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0097_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0098_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0099_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0100_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0101_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0102_sourcesDetected.fits - cannot calculate astrometry
   WARNING - No REFCAT in S20170505S0103_sourcesDetected.fits - cannot calculate astrometry

Display the stacked image¶

The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the "SCI" extension, the variance is in the "VAR" extension, and the data quality plane (mask) is in the "DQ" extension.

In [11]:
image_file = 'S20170505S0095_image.fits'
hdu_list = fits.open(image_file)
wcs = WCS(hdu_list[1].header)
hdu_list.info()

Filename: S20170505S0095_image.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     321   ()      
  1  SCI           1 ImageHDU        90   (4275, 4262)   float32   
  2  VAR           1 ImageHDU        90   (4275, 4262)   float32   
  3  DQ            1 ImageHDU        90   (4275, 4262)   int16 (rescales to uint16)   
  4  PROVENANCE    1 BinTableHDU     17   10R x 4C   [28A, 128A, 128A, 128A]   
  5  PROVHISTORY    1 BinTableHDU     17   19R x 4C   [128A, 428A, 28A, 28A]
In [12]:
image_data = fits.getdata(image_file, ext=1)
print(image_data.shape)

(4262, 4275)
In [13]:
plt.figure(figsize = (15,15))
plt.subplot(projection=wcs)
plt.imshow(image_data,cmap='bone',norm=Normalize(vmin=1, vmax=1000),origin='lower')
plt.xlabel('Right Ascension [hh:mm:ss]',fontsize=14,fontweight='bold')
plt.ylabel('Declination [degree]',fontsize=14,fontweight='bold')
plt.show()

Optional: remove duplicate calibrations and remove raw data (uncomment lines before running)¶

In [15]:
# %%bash

# cp S20170505S0095_image.fits final.fits
# rm -r calibrations
# rm gsaoi.list gsaoi_data_reduction.log S2017*
# mv final.fits S20170505S0095_image.fits
In [ ]: